Skip to content

Fix: MoE Memory Budgeting and Dynamic Multimodal Tokens - #98

Closed
roydsouza wants to merge 2 commits into
SharpAI:mainfrom
roydsouza:fix/moe-memory-and-multimodal-tokens
Closed

Fix: MoE Memory Budgeting and Dynamic Multimodal Tokens#98
roydsouza wants to merge 2 commits into
SharpAI:mainfrom
roydsouza:fix/moe-memory-and-multimodal-tokens

Conversation

@roydsouza

Copy link
Copy Markdown
Contributor

Description

This PR addresses two critical issues identified during an adversarial audit of SwiftLM on Apple Silicon (M5):

  1. MoE-Aware Memory Budgeting: Previously, MoE expert weights were not accounted for in the physical RAM budget when using SSD streaming. This led to 'swap-storms' as active experts were paged in over the limit. This PR adds a 2GB safety buffer for MoE models and includes a high-swap usage monitor.
  2. Dynamic Multimodal Token Resolution: Resolved hardcoded boaToken (255010) and eoaToken (255011) by extracting them dynamically from config.json. This fixes expert routing for non-Qwen multimodal models.

Changes

  • Added moeBuffer to computeSSDMemoryBudget.
  • Integrated sysctl vm.swapusage check in Server.swift.
  • Implemented extractMultimodalTokens in OmniModelFactory to resolve BOA/EOA tokens from config.

Verified on Apple Silicon M5 with Gemma 4 MoE.

@solderzzc

Copy link
Copy Markdown
Member

Thanks @roydsouza — and credit where due: the multimodal BOA/EOA half of this landed via your #104 in May, and it has been on main with tests since (9d495d9, 621a931). The MoE-detection half is now superseded by #114, which extends the same idea to all three expert-key spellings (num_local_experts / num_experts / n_routed_experts, plus nested text_config).

What remains unique here is the MoE memory buffer and the swap-usage warning, and both are worth having. One fix needed on the swap monitor: the hand-declared xsw_usage struct has xsu_used / xsu_avail in the wrong order relative to Darwin's sys/sysctl.h (real layout is xsu_total; xsu_avail; xsu_used;), so the warning fires on available rather than used swap — using Darwin's imported xsw_usage directly avoids both the bug and the type shadowing.

This branch also carries an accidental working-tree state that reverts ~2,300 lines of unrelated shipped code (speculative-decoding dispatch, ModelDownloader, SettingsView, several safety gates), so rather than rebasing it, a fresh small PR on current main with just those ~40 lines — the computeSSDMemoryBudget MoE buffer and the corrected swap warning — would be the clean path. Happy to help set that up if useful.

Closing in favour of the pieces that already landed (#104, #114). The buffer + swap-warning invitation stands.

@solderzzc solderzzc closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants